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ABSTHACT 

This paper discusses those generic features of school 
districts as complex organizations which inhibit district 
coordination of testing, evaluation and instruction. It is based on a 
selective literature review and four theoretical papers covering (1) 
school district decision making, (2) the institutional structure of 
school districts, (3) role conflict among research and development 
directors, and (4) the schism dividing achievement monitoring from 
analytic evaluation. Analysis of these field study data revealed the 
salient characteristics of school districts as complex organizations 
and the implications these attributes have for the focus of this 
research. The sections are organized as follows: introduction; 
generic factors which inhibit district linkage of testing, 
evaluation, and instruction; situation-specific conditions which can 
override those generic factors; variations observed among the school 
districts; and conclusions that may be helpful to district managers 
wanting to develop management strategies to coordinate their own 
testing, evaluation and instruction operations. (Author/PN) 
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For the last two years, we have examined the problem of how school 
districts can effectively link testing and evaluation activities to their 
own on-going Instructional programs. 

We became curious about this question because most school districts 
seemed to be ignoring the potential testing and evaluation data have for 
instructional improvement. Many districts have considerable capacity to 
collect and analyze test data; the evaluation community has emphasized a 
multi-disciplined approach to utilization-focused evaluation. Few dis- 
tricts seemed to have incorporated these activities and developments int 
their regular, ongoing decisionnmaking procedures. 

Why don't district administrators do what seems so obvious to many 
outsiders — that is, analyze test scores and other evaluative data in a 
systematic way to find out what students have already learned and then make 
appropriate inferences about ways to improve the districts' instructional 
programs so as to enhance students' learning? 

We suggest that even if technical difficulties were overcome, the 
generic characteristics of school districts make this phenomenon relatively 
rare. But we have discovered conditions and activities in some districts 
that have enabled them to forge this linkage among testing, evaluation and 
instruction. We believe that other districts can learn from their experi- 
ence and forge similar linkages. 

In this chapter, after a short introduction we discuss those generic 
features of school district as complex organizations which inhibit district 
coordination of testing, evaluation and instruction. Then we analyze spe- 
cific conditions in our sample districts and offer some conclusions which 
might be of use to interested school districts. The sections are organized 
as follows: 

Introduction 

Generic factors which inhibit district linkage of test- 
ing, evaluation and instruction; 

Situation-specific conditions which can override those 
generic factors; 

^ Variations we observed among the six districts; 

^ Conclusions that may be helpful to district managers 
wanting to develop management strategies to coordinate 
their own testing, evaluation and instruction operations. 
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INTRODUCTION 

In 1975, the Center for the Study of Evaluation (CSE) at UCLA, submit- 
ted to the National Institute of Education (NIE) a long-term plan for an 
integrated set of studies focused on school districts' use of evaluation. 
At that time when many national policy-makers mandated evaluations of large 
scale federally funded programs for their, the policy makers', decision- 
making purposes, we urged that the local district decision-making uses of 
evaluation also deserved attention. The needs of legislators and admini- 
strators for information about the effectiveness of those educational inno- 
vations supported by external funding was matched, we felt, by the needs of 
district boards and administrators for information about teaching and 
learning the presumably routine business of schools districts supported 
by local dollars. 

Prior to the 1960's, teachers generally administered and marked 
quizzes or tests they had developed themselves or they used tests provided 
by text publishers. Teachers used these tests primarily to motivate stu- 
dents to study, to give grades, and to check up on who was learning and who 
wasn't. Guidance and counseling personnel also administered tests as part 
of their diagnostic procedures for placing children in special programs or 
referring them for additional services. 

However, by the mid 1960's, with the advent of evaluation requirements 
that accompanied externally funded programs, school district central 
offices became involved in extensive testing and evaluation activities. 
Thus, teachers began to also administer standardized tests in addition to 
their own classroom tests. 
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By 1975, many district offices began to test all children, not for 
grading or class placement purposes, but to evaluate district-administered 
but externally funded programs; not for reporting to parents about their 
own children, but for sending reports to distant offices so that legisla- 
tors and administrators could decide whether to continue, modify or stop 
particular programs. 

Each school district organized its response to those external demands 
for information in its own way. Some districts hired consultants who wrote 
the required evaluation reports by culling data from routinely administered 
district-wide standardized test data. Other districts attached in-house 
evaluators to each of their categorically funded programs — so that bi- 
lingual evaluator, special education evaluator, compensatory education 
evaluator became common job descriptions. Still other districts, especi- 
ally the larger ones, consolidated testing and evaluation into organized 
units called by various combinations of words such as "testing," 
"evaluation," "assessment," "policy," "planning," and "research." 

Concomitant with, and in support of district testing and evaluation 
activities, university-based evaluation activities also expanded. Within 
ten or fifteen years, evaluation research became an identifiable profession 
replete with its own division within the American Educational Research 
Association, university graduate training programs, and journals. Profes- 
sional recognition was bestowed when the Library of Congress listed evalu- 
ation research as an official catalogue description in the late 1970's. 
The field expanded conceptually as well as numerically. Academic evalua- 
tors moved from a narrowly focused concern with forcing an experimental 
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research paradigm onto real life settings towards realizing that evaluation 
methodology had to match the special demands of the situation and the eval- 
uation audience. 

It was within this context that in 1975 the UCLA Center for the Study 
of Evaluation (CSE) proposed to the National Institute of Education (NIE) a 
multi-year research studly of how school districts were conducting their 
testing and evaluation activities. Our research was divided into two 
parts: 

to describe, by means of a national survey, how school districts 
organized, staffed, and funded their evaluation units; 

^ to examine, by case studies, school districts who were attempting 
to use evaluation and testing to serve local instructional 
decision-making purposes. 

The results of the national survey have been reported elsewhere. 1 
^ An important focus for our case studies was to explore generic school 

district organizational characteristics and try to determine which charac- 
teristics affected the ways districts used tests and evaluations for in- 
structional decision making. We wanted to understand if and how school 
district generic characteristics affected the ways school districts managed 
their testing, evaluation and instruction activities. District central 
office administrators appear to have considerable discretion when they 
design their districts' testing and evaluation activities, i.e., to select 

n Lyon, C. D., Doscher, L., McGranahan, P., & Williams, R. C. 
Evaluation and school districts . Los Angeles: Center for the 
Study of Evaluation, University of California, 1978. and Williams, 
R. C. Studies in evaluation and decision making: Evaluation 
offices in small school districts. CSE Report to the National 
Institute Education . Los Angeles: Center for the Study of 
Evaluation, University of California, November 1979. 
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texts, within state regulations; to standardize curricular scope and se- 
quence within and across grade level, and to direct teachers to use various 
classroom management and instructional practices. We wanted to understand 
those organizational influences that seem to inhibit the districts' 
coordinatir.g their testing and evaluation with instructional decision 
making. 

Our project had two parallel thrusts. One was understand school 
districts as complex organizations and the other was to conduct field stu- 
dies. We conducted our own selective review-of-t he-literature on school 
district organizational characteristics. In a parallel activity, we com- 
missioned four outstanding organizational theorists from the University of 
California to help us. Each theorist conducted a separate analysis of the 
data from CSE's national survey of school district research and development 
units and prepared a paper. Each of the four papers reflected the author's 
particular theoretical interest and orientation. The topics covered were: 
school district decision making, the institutional structure of school dis- 
tricts, role conflict among R&D directors and the schism dividing achieve- 
ment monitoring from analytic evaluation. 2 

Armed with our own organizational literature revV^iW and the four ana- 
lytical papers, we conducted field studies over a two year period in se- 
veral districts that were reputed to have a management strategy for linking 
testing and evaluation with their instructional decision making. Using a 

2. These papers have been published as part of the CSE Monograph Series in 
Evaluation with the title Evaluation in School Districts: Organiza- 
tional Perspectives , edi ted by Adrianne Bank and Richard C. Williams, 
CSE, 1981. 
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reputational approach, we found about 40 districts that at least someone 
identified as making an effort to link their testing and evaluation with 
instructional decision making. From among these nominees we identified 
through telephone screening and visits, several districts whose actual 
activities measured up to their reputation. Our final set of districts 
were diverse in: size, location and ethnicity. All of our final districts 
which had operated their management system for linking testing and evalua- 
tion with instruction for a number of years. 

Examples of what these districts were doing to link testing and evalu- 
ation with instruction are as follows: 

One small district had first constructed a K-6 scope and sequence of 
skills for math, language arts and reading, then had teacher teams develop 
criterion referenced tests (CRT's) for selected milestone objectives. 
Students now take these CRT's quarterly. The results are used by teachers 
within classrooms to regroup students for Instruction and by the district 
for long-range instructional program development and revision. 

In another larger district the evaluation unit provided individual 
schools with composite data from norm-referenced tests and parent/teacher 
surveys so that school site planning teams could develop budgets and action 
plans for the coming year. 

The central office staff in p third district concentrated on develop- 
ing mini -courses and workshops for a staff development program. The con- 
tent of the courses came from the district's annual analyses of students' 
performance on state assessment tests. 
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A fourth district office used student test score data comparing 
students in two different instructional modalities to support their 
efforts to transform all elementary schools into "fundamental" schools. 
Teacher compliance with the district mandate and student achievement was 
monitored by the district using classroom observations and student test 
scores on norm-referenced tests. 

In yet another district, the central administration had developed, 
with extensive teacher and principal involvement: a district instructional 
continuum, desired teaching methodologies, supporting criterion-referenced 
tests and a supervisory system for carefully monitoring the program's 
coordination and implementation. 

In each district we spent several person-weeks observing, reading 
documents, interviewing board members, administrators, principals and 
teachers and visiting schools and district units. We sought answers to 
such questions as: why is this district developing management strategies 
to link testing and evaluation with instruction when so many others are 
not; what specific conditions, processes and techniques have marked their 
efforts; what have been the results of their efforts? 

GENERIC FACTORS INHIBITING DISTRICT LINKAGE OF TESTING, EVALUATION AND 
INSTRUCTION 

Based on our own literature reviews and the faculty papers that were 
produced from this study, we developed the following perspective on the 
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salient characteristics of school districts as complex organizations and 
the implications these attributes have for the focus of our research. 

Organizations can be described as groups of two or more people who 
have come together to accomplish goals that could not be accomplished indi- 
vidually. For many years, theoreticians and practitioners have been trying 
to classify the characteristics of organizations in order to better under- 
stand their functioning and thereby improve their effectiveness and effi- 
ciency. The search for a common set of management principles or an all- 
encompassing theory of organisations has proven to be difficult and com 
plex. Somehow the deeper one probes into the dynamics of organizations, 
the more one realizes how much there is to learn. At the present time the 
field of organizational theory is in a healthy state of disarray. Rather 
than continuing to search for the a1 1 -encompassing theory, many researchers 
have tried to distill and describe the essential features of different 
kinds of organizations. By so doing they have hoped to gain a better 
understanding of specific types of organizations and of organizations that 
share common properties. s 

For example, San Quentin Prison, IBM and the Roman Catholic Church are 
complex organizations. But even a cursory examination will reveal vast 
differences among them regarding characteristics such as: reasons for mem- 
bership in the organization, selectivity of clients, eiuthority structures, 
agreed-upon measures of success. We are likely to make more progress in 
understanding the management strategies used in complex organizations if we 
separate organizations into appropriate classifications, and compare 
examples which exhibit common critical characteristics and are thus in the 
same category. 

ER?C 10 
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For purposes of this analysis we have chosen four important generic 
characteristics of typical public school districts that we feel make it 
difficult or provide disincentives for districts to link their instruc- 
tional decision making to district testing and evaluation victi vities : 

" institutionalization and domestication 

" goal diffuseness 

" permeable boundaries 

" loose coupling 

In the following paragraphs we will describe each of these characteristics 
and discuss their implications for school district linking activities. 
Districts as Domestic Institutional Educational Organizations 

When we use the phrase "domestic and institutional" (Carlson, 1964; 
1965), we mean to imply that school districts are organizations upon which 
have been placedi many "Wernal societal controls. These external expecta- 
tions, regulations and conitcaints leave district managers with little 
autonotny in certain matters. For example, school districts cannot, for the 
most part, choose their clients, i.e,, students. These clients or stu- 
dents, in turn, are for the most part obligated to attend schools run by 
the district. Some children, of course, can choose to attend private 
schools, but the percentage who have done so in 'the past has been quite 
small in most communities and remains only a small percentage of the total 
school age population. The school district's level of financial support is 
externally determined and only minimally guaranteed. Typically the school 
site's support level does not vary with the schools' performance (Carlson, 
1964; Gallaher, 1965), but in accordance with financial and political 



ERIC 



- 10 - 



considerations beyond the district's power to influence. Additionally, 
teachers' unions and contracts have limited the power of administrators to 
hire, fire, and transfer personnel. 

One consequence of this domestication is that the school district need 
not "prove their worth" in the marketplace on any regular or critical ba- 
sis. That is, unlike many private sector organizations, their continued 
flow of both clients and funds is relativv^ly Independent of the quality of 
service. Consequently, the quality of the district's technical services — 
i.e., instruction — m<^y receive less sustained and system-wide attention 
from managers than such i^ervices would in some other organizations, either 
public or private, where the critical relationship between the success of 
the technical core and the organization's survival is more clearly evi- 
dent. What does receive sustained attention from school district managers 
are those areas in which the school is held publicly accountable — e.g., 
honoring union contracts, keeping schools open a minimum number of days, 
complying with health, safety and financial regulations, etc. 

Viewed from this perspective, there does not appear to be a strong 
"bottom line" incentive for schools districts to focus on improving the 
instructional technical core particularly through using test scores and 
evaluative data to monitor student outcomes and program inputs (Meyer, 
1977; Niedermeyer, 1979). Indeed, some (Meyer and Rowan, 1978) argue that 
it may be detrimental to their public survival for school districts to link 
test scores and evaluative data to instructional improvement efforts in any 
highly visible way. They assert that schools, like libraries and national 
parks, are institutionally valued. Therefore, their constituents, the 
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public, regard them as having instrinsic worth rather than worth derived 
from market-place success or empirical demonstrations of their technical 
competence. The public seems to have believed — at least until very 
recently — that schools have high social worth. This being the case, 
school districts have not been called upon to demonstrate their technical 
worth in terms of student achievement. These authors argue that if the 
public were given the means to clearly link student test scores with the 
instructional program, this might prompt many to recognize that schools do. 
have, in fact, a weak technical core and have limited capacity to provide 
Instruction appropriate to meet student needs. This could cause a serious 
breach in confidence, precipitating a decline in public support, both in 
terms of dollars and pupils. If such an argument is valid, and we believe 
it is, it is likely that many school district central offices have little 
incentive to develop a highly visible system linking testing and student 
test scores or evaluative data about educational activities to a district- 
wide instructional improvement effort. 

Goal Diffuseness . Likewise, school districts have difficulty in de- 
fining instructional goals such that parents and citizens can clearly iden- 
tify high priority learning objectives and the extent to which the students 
are achieving them. Some observers have noted that although there may be 
general agreement on the basic skills which our society expects schools to 
provide to students (Goodlad, 1979), there are national and local disagree- 
ments as to which basic skills need the greatest immediate emphasis, or, 
beyond the basic skills, which subjects have the highest priority 
(El boim-Dror, 1970). . For example, various publics have different views 
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about the importance of the schools' role in developing critical thinking 
skills, in providing students with a critical sense of historical events, 
and in teaching students to appreciate art, literature, culture? The plu- 
ralism of our society tends to introduce conflicts about instructional 
goals and priorities which some'school districts may resolve by leaving 
goals and objectives ambiguous and general. 

Interestingly, there is less ambiguity about the intended outcomes of 
the many non-instructional activities and responsibilities which have been 
assigned by public or government agencies to schools over the years -- for 
nutrition, medical services, and counseling, for example. In addition, 
interest groups with some success have pressured the schools to allocate 
additional instructional resources, often specified in great detail, to 
selected categories of students, e.g., economically deprived, handicapped, 
bilingual, gifted. Thus, school district staffs seem to have neither the 
independence, authority, nor the means to order their own internal priori- 
ties and then set about a carefully planned, long term course of action to 
achieve clearly specified instructional goals (Elboin Drop, 1970). This 
goal diffuseness around the district's and the schools' core instructional 
activities supports our earlier observation that the survival of schools 
and districts as organizations seems to depend more on external accommoda-- 

tion than on goal attainment. 

Given the importance of external accommodation, schools tend to de- 
pend on ceremonial events as evidence of school effectiveness (Meyer and 
Rowan, 1978; Weisbord, 1978). For example, the instructional problems re- 
sulting from a high concentration of low-income pupils in a school may be 
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resolved by providing additional funds and other resources to those stu- 
dents with the announced expectation that more money somehow will "make a 
difference." Few resources within the district are devoted to researching 
or analyzing the way alternative instructional strategies might remedy the 
pupils' academic deficiencies. Public attention is instead directed by 
school personnel towards the extensi veness of the districts' symbolic con-- 
formity, e.g., the districts' use of approved texts, the number of creden- 
tialled professionals, compliance with state curricular mandates, even 
though there may be little evidence that these factors make any substantial 
difference in student learning. 

We have argued that school districts have up until now been largely 
successful in protecting their instructional or technical activities from 
close public scrutiny or criticism even though the public has long had 
influence over other vital organizational matters, especially capital out- 
lays and student regulation. In these matters, Lieberman (1977) has ob- 
served that, at the local level, schools are among America's most publicly 
influenced institutions. School district policies have been controlled by 
state laws and regulations. And school districts have become dependent on 
federal and state financing to aid them in addressing what are considered 
pressing social problems, e.g., reducing racial isolation, mainstreaming 
handicapped pupils, etc. Compliance indicators almost always accompany 
such funds, thus lessening local discretion (Halperin, 1980; Kirst, 1980). 
Permeable Boundaries 

From an open systems viewpoint (Katz and Kahn, 1978; Rogers, V., and 
Rogers Argarwala, R., 1976), school districts can be viewed as engaging in 
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a continuing interaction with many proximal and distal environments, e.g., 
parents, community, state and federal agencies. School districts, as 
organizations, indeed seem to have very permeable boundaries (Pfeffer & 
Salancik, 1978). 

Districts differ, of course, with regard to how they are affected by 
this boundary penetration. Some communities, schools and district person- 
nel are left relatively unnoticed to do their jobs in whatever ways they 
see fit. In other coiranunities there may be considerable heat around parti- 
cular instructional issues. Controversy over school issues may come and 
go. The federal and state impact on schools is similarly varied. Some 
small districts have very few students who are the recipients of federal or 
state attention. Consequently, these districts remain relatively free of 
this type of external regulation. Other districts, larger or in urban 
settings, have many such pupils and therefore receive considerable funding 
— and experience considerable outside regulation which may begin to impact 
on their instructional programs. 

And, the type of federal regulation differs from situation to situa- 
tion. For example, federal regulations have sometimes required that norm- 
referenced test instruments be used as the means of documenting program 
effectiveness. On other occasions, courts have outlined the specific na- 
ture of the test data acceptable as evidence of district competency in pro- 
viding equality of educational opportunity. Given the power of the legis- 
lative, executive and judicial branches of both federal and state govern- 
ments to impose sanctions for lack of compliance (e.g., withdrawal of 
funds, court-ordered busing) some school districts are finding it necessary 
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to publicly discuss and justify the effectiveness of their technical acti- 
vities — that is, their instructional program. 

This breaching of school district boundaries by federal, state and 
local groups in the area of instruction is a relatively new phenomenon. 
Not all observers view this trend as necessarily ameliorative or logical 
(Wise, 1977). It does seem possible, however, that such penetration, if it 
continues, may result in major changes in the internal functioning of 
schools and districts, especially in relation to characteristics such as 
goal diffuseness and loose coordination. 
Loose Coupling 

A current and widely held view of the internal operations of public 
school districts is that, as compared with some other bureaucracies, they 
are loosely coupled (Weick, 1976; Meyer and Rowan, 1978). This means that 
control, supervision and sanction linkages among individuals operating at 
the various levels of the system — central administrators, principals, 
classroom teachers — are weak. The intent and meaning of messages may 
become diffused or altered as they pass up or down through the various 
administrative levels. As a result, directives, orders or regulations from 
the central administration may be ignored, misapplied, or only partially 
implemented at lower levels without serious consequences. 

Loose coupling is a useful metaphor for describing the authority chain 
in public school districts. But the work of several researchers 
(Abramowitz, 1978; Daft and Becker, 1978; Downs and Mohe, 1976; Lortie, 
1977; Spence et al., 1978) suggests that the loose coupling designation 
should be applied differentially and selectively to particular aspects of 
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school district functioning. That is, administration in those areas of 
public concern which are objects of regulation seem to be tightly coupled 
and policies are enforced very directly and effectively. For example, cen- 
tral administati ve policies regarding pupil control (e.g., use of corporal 
punishment) or school scheduling (e.g., beginning and ending times for the 
school day) are usually closely monitored and fully implemented in all 
schools and classrooms. 

However, in areas hitherto exempt from public regulation, e.g., teach- 
ing strategies or classroom management, loose administrative coupling 
appears common (Firestone, Herriott, Meyer, et al., 1978; Meyer and Rowan, 
1978). Central administrators rely on "confidence and good faith" with 
regard to teachers' instructional decision making. The tradition of teach- 
er autonomy (Lortie, 1975), reinforced by increasing teacher union power, 
weakens the ability of central administrators to influence teachers' every- 
day classroom behavior and instructional decision making. Schools and 
classrooms, in terms of instruction, can be thought of as being federations 
or collections of zones of influence (Lortie, 1977; Miles, 1980). Decisions 
about technical matters — i.e., instruction -- rather than being centrally 
determined, widely disseminated and carefully enforced are, instead, par- 
celled out to those who have the immediate and direct responsibility for 

action. 

To summarize: It is our view that school districts can be character- 
ized as organizations in which there is a weak technical core, that is, 
only a small body of professional knowledge regarding how to cause student 
learning. The fact that instruction is not a certain science but an 
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uncertain craft is usually hidden from public view* The school district's 
organizational boundaries are easily breached by outside public agencies 
and groups each of whom makes what is assumed to be a legitimate claim to 
influence policy and practice. In general, school districts operate with 
diffuse goals which do not function as internal planning guides to tailor 
policy, but rather are external symbols, demonstrating to the public the 
institution's value. Central administrative decision-making units especi- 
ally in the technical -core instructional areas function with a great deal 
of autonomy from one another. 

It is no wonder, then, that management strategies to coordinate test- 
ing and evaluation activities with instruction do not naturally occur with- 
in school districts even when we take into account the limitations imposed 
by the technical problems of test selection or development, analysis and 
interpretation. Generic organizational characteristics resulting from 
school districts* relationship to their external environments act as dis- 
incentives inhibiting the internal use of testing and evaluation for in- 
structional improvement. Thus, it is not surprising to find that even dis- 
tricts which spend considerable time and resources to collect test data 
about student achievement for shipment to outside agencies do not analyze 
these data in any systematic way so as to improve their own instructional 
program . Similarly, districts which produce evaluations of district 
programs often do not use these evaluative. studies as ways to improve pro- 
grams, but, rather as signals to outside agencies that they are in compli- 
ance with regulations (Zucker, in Bank & Williams, 1982). 
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It follows that those school district administrators and policy makers 
who have forged a system-wide testing/evaluation/instruction linkage have 
contended with these generic characteristics in some manner, 

SITUATION SPECIFIC CONDITIONS WHICH OVERRIDE GENERIC FACTORS 

Given these generic characteristics of school districts, one might 
assume that no districts would be linking their testing and evaluation 
activities with their instructional decision making. Yet, we were able to 
identify a sample of districts that were clearly trying to forge such a 
linkage. Why? What were the conditions or characteristics of these 
districts that seemed to account for their overcoming these generic 
inhibitions? 

What were the specifics around these factors that have allowed some 
school districts to develop management systems that link their testing pro- 
grams with classroom instruction? Why do some districts in addition to 
using student test scores as evidence of student learning use other evalua- 
tive data such as parent, teacher and student opinions and questionnaires 
to identify educational Inputs which may be related to outcomes? Why are 
they Interested in providing to teachers, principals, planning teams or 
central office administrators information which is of use in modifying pre- 
sent instructional practice so as to enhance future learning? Although 
each of the school districts we studied was unique, all of our "heroic" 
districts exhibited characteristics in common, which, in combination, were 
sufficient to help them overcome natural obstacles. 
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All the districts we studied indicated that a major reason lor begin- 
ning to coordinate tests and instruction was their desire to raise pupil 
achievement as measured by test scores. District officials indicated that 
test scores were considered as a surrogate description of student achieve- 
ment. They believed that testing and evaluations using test data could 
then be used to guide them in making informed instructional decisions at 
the school site or district level. 

Their direction and activities can be explained by what are considered 
incentives, that is, unique internal or external pressures or drives that 
moved them to act. For example, some central offices seemed moved in this 
direction by explicit mandate from courts^ or from state legislatures or 
from school boards. In other districts, superintendents or other officials 
reported that they were influenced by research or university thinking, and 
decided to use available federal and state money to build their own in- 
structional ly relevant tests. In many of the districts there had been 
expressions of dissatisfaction, coming from parents and teachers, with the 
academic performance of students. Thus, in one district, administrators 
first started to use test scores to identify students needing remedial 
help. In another district, however, there was general parent and teacher 
satisfaction with student learning. However, the district superintendent 
and others in the central office felt that a system of individualized 
instruction might further increase the learning of average and above 
average college-bound students. 

We can categorize the types of incentives we found as either external 
or internal to the district central office, as in Figure 1. 
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Figure 1 



Types of Incentives 
External to the District Internal to the District 



® requirements by federal or 
state legislative, executive 
or judicial agencies to: 

- evaluate programs 

~ develop courses of study 

- raise test scores 



availability of federal and 
state money for: 

- text development 

- evaluation of programs 

- staff development 

- relationships with 
universities 



® community dissatisfaction with 
public education expressed 
through: 

- press and media 
« loss of students 

- Board action 



^ belief by administrators that 
instruction could impact test 
scores a "vision" of 
enhanced student achievement 



^ district desires to acquire 
additional funds 



knowledge by administrators 
of technical aspects of 
testing and evaluation 



desire for improved district 
image in the eyes of the 
community^ profession 



^ existence of techniques or 
procedures to analyze test 
in an inSstructional ly relevant 
relevant manner 
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But our list of incentives, to some extent, begs the question, since . 
these are often found in other districts as well. We were still uncertain 
as to why some districts reacted to these incentives by moving to link 



Given our small sample, and our field-based research design, we cannot 
provide a definitive answer to that question. What we feel is reasonable 
to assert, however, is that particular internal conditions seemed to be 
present in each of our six districts. Each of our districts developed a 
particular vision of how to respond to a challenging situation and as- 
sembled the management capacity to make it happen. Four internal condi- 
tions or characteristics exist across districts which seemed to be criti- 
cal; they were: idea champions (Daft & Becker, 1978), stable core staff, 
comprehensive problem analysis, and tolerance for delay and ambiguity., 

^ Idea champions by this we mean individuals in key 
administrative and policy positions who firmly believed 
In some variation of the following position: that class- 
room instruction could be conceptualized in terms of stu- 
dent learning outcomes, that tests of student learning 
outcomes and evaluations of instructional activities 
could be used to relate instruction to learning, and that 
district policies and procedures should be attentive to 
this linkage. 



testing and evaluation with instruction when others didn't. 



In our districts, the presence and constitution of these 



idea champions were essentially variations on a common 
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theme. The common theme was the emergence of a critical 
mass of individuals who believed in the power and poten- 
tiality of their linking process. The variations related 
to their evolution and make-up. These critical masses of 
individuals generally started with one person and a small 
but eventually growing cadre of followers. The organiza- 
tional role of the person with this initial vision varied 
— some were central administrators with varying respon- 
sibilities, some in testing, some in curriculum, some 
were central Tine administrators. Soon this critical 
mass grew to a point where it included influentials in 
all of these roles — and this group engaged the support 
and confidence of the top central administration and 
school board. 

Stable core staff — in car districts, this critical mass 
of "idea champions" and their followers had been around 
for a while. In those districts with the most advanced 
linkage systems, it took more than five years for the 
linkage programs to develop and mature. A stable core 
group even with some additions or deletions, continued to 
work rather steadily at the process over several years. 
They were thus able to "ride out" the shifts in district 
directions that sometimes accompany shifts in school 
board memberships or other exogenous shocks (e.g., court 
orders, changes in funding levels). 
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Comprehensive rather than ad hoc problem analysis on the 
part of district managers. The core staff stayed aware 
of next steps beyond the immediate task of the moment. 
Typically our district core groups did not develop elabo- 
rate, multi-year bluprints of their projected activi- 
ties. They did have a comprehensive "sense" of where 
they were going and the steps that were necessary to get 
there. It is one set of tasks to develop, for example, a 
CRT program in reading — it is quite another set of 
tasks to actually get teachers to use it. Bridging the 
gap between development and use implies an understanding 
of the school site and district as a social system and an 
appreciation of the various strategies and tools that 
might be most effective. 

Tolerance for delsty and ambiguity — the personnel within 
these districts were aware of, and accomodated to a de- 
velopment strategy that was uneven, interrupted, and in 
some cases often postponed^.. ^ Components of the linking 
system often developed independently of one another; each 
sometimes had different purposes and each had its own set 
of advocates. Creating arrangements to merge together 
these disparate pieces into new configurations took time, 
patience, administrative talents, and adherence to the 
overall "idea" rather than to specific details. 
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The above list is not exhaustive. It may be that these same 
characteristics and activities are found in many districts that have not 
tried or who have tried and abandoned efforts to develop a linkage system 
between tests, evaluation and instruction* All we can say at this juncture 
is that these characteristics are present in our sample districts, and dis- 
trict administrators believe these characteristics contributed to their 
districts' progress. 

Goal s . As we noted previously, districts often adjust to their orga- 
nizational boundary permeabil ity by promulgating diffuse or general goals^ 
Although such goals may function well symbolically, their lack of sped ft 
city makes it difficult for district managers to formulate long term 
plans. Our six districts did not differ from the general case, here, in 
terms of clearly stated goals for developing their system. Ultimately, 
however, some, but not all, of them emerged with clearly stated goals for 
instruction. 

Rather than developing a carefully sequenced, long-term plan of 
action, our districts had a general vision towards which they were 
driving. When appropriate, they changed the specific activities and proce- 
dures, but the general vision remained before them. All began with the 
idea of instruction conceptualized in terms of student learning outcomes 
and stressed this in a variety of ways. 

They legitimized and shaped Informal discussions as well as policy 
decisions around the Idea of student learnings which could be tested, and 
around educational activities which could be described. 
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They found allies for. such an instructional orientation among opinion 
leaders within the central office and teachers and principals. 

By presenting a vision to various subgroups, they raised morale, and 
transformed feelings of helplessness into feelings of empowerment. 

They reinterpreted the district's past and present activities in the 
areas of testing, evaluation and instruction so as to provide the rationale 
for future activities relating to a more comprehensive data-based instruc- 
tional improvement system. This process of reordering some of what had 
already been done was especially noticable in districts* writing of propo- 
sals for new funds. 

The overall planning for a data-based management system that occurred 
was akin to that described by Lotto, Clark and Caroll (1980). 

"Conceptually, planning is not a "synthetic" management function but 
an essential part of the way in which individuals in organizations make 
sense of and create their organizational reality." 

"Operationally, all organizational participants should be considered 
to be active planners. Consequently, those individuals assigned formal 
administrative responsibilities for planning should view themselves less as 
managers, initiators, and monitors and more as stimulators, facilitators, 
and orchestrators." 

Administrative coupling . In many districts, units which carry out 
functions relating to curriculum, instruction, supervision, administration, 
personnel, and budgeting often operate semi -autonomously. That is, staff 
members within each of these operating units rarely have formal 
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opportunities to share their skills and expertise with one another in 
relation to overall instructional improvement policies. This is especially 
true in large districts 

The same observation can be made about teachers that is, that they 
work autonomously behind closed classroom doors (Lortie, 1975). To be 
sure, teachers ma(y be visited by principals and other supervisors, espe- 
cially during their probationary term, but for the most part teachers are 
largely unsupervised when it comes to the day-to-day interactions they have 
with their pupils. Usually, the relationships between each teacher's 
methods and his/her pupils* learning, are largely unkf . ' can at best 
only be estimated (Goodlad, 1974). In many districts «e have a curious 
situation where what would seem to be the most important interaction in a 
schooling system, that is, teacher classroom behavior and its effects on 
pupils' learning, is largely unattended to in any direct supervisory manner 
by the central district office. Most school districts, when it comes to 
managing their most important interactions — namely the interaction be- 
tween teachers and pupils and the effects thereof — are loosely coupled 
(Meyer, 1977- Weick, 1976). 

In our six districts, however, coordinating mechanisms were developed 
which horizontally linked central office units units into a management team 
and which vertcally linked central offices with schools and classrooms. 

Several types of coordinating mechanisms were used. In smaller dis- 
tricts, central office staff had informal and social relationships with one 
another — going to lunch together, talking in the corridors and between 
offices. In larger districts, staff meetings, ad hoc task forces. 
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cabinets, councils, and other types of working groups were used to link 
central office operations. 

In some districts, strong linkages were established between operations 
at the district level. In others 1ntra-school coordination coordination 
was stressed. For example, in one district, the central office staff de- 
velopment unit worked very closely with the testing unit. Analyses of stu- 
dent test scores were quickly translated into staff development programs 
for defined groups of teachers. But, at least during the early years in 
this district, curriculum people were not brought into the discussion. In 
another district, each school had a media specialist and a learning specia- 
lists who Interacted on an ongoing basis to coordinate assistance to teach- 
ers in grouping students for Instruction, providing them with remediation, 
enrichment texts or worksheets. The principal routinely visited classrooms 
and discussed classroom management with teachers. 

In our large district, management of the entire district was regulated 
by instructional considerations. Principals supervised teachers in rela- 
tion to pre-specified district-wide "Elements of Quality" which were expec- 
ted to be in evidence In each classroom. Principals in turn were super- 
vised by central office personnel In terms of the extent to which teachers 
were adhering to district processes. Test scores, both criterion-refer- 
enced and norm-referenced, were used to monitor student progress. On an 
annual basis, areas which students appeared not to be learning were 
Identified; analyses of instruction, texts, teaching strategies were then 
made to determine if action should be taken. 
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VARIATIONS AMONG THE SIX DISTRICTS 



Briefly, we have alluded to factors which were present in districts 
which developed an information-based instructional management system. Now, 
we will describe some of the dimensions on which the districts varied. 
District management systems differed from one another along at least three 
dimensions: purpose, extent of coordination, and locus of decision-making. 
Purpose of System: Narrowly or Broadly Defined 

Some districts narrowly defined the purpose of their system as RAISING 
STUDENT TEST SCORES. Their purpose was to raise overall student test per- 
formance or the performance of particular subgroups. In these districts, 
some of the following activities to link testing and instruction were 
undertaken: 

® modifying instruction to fit whatever tests were being 
used, that is 

- analyzing the current tests by the domains of know- 
ledge skills represented by the items 

- indexing textbook content to test content 

- increasing instructional time on tested skills 

- developing practice materials for tested skills 

- teaching to test items 

finding tests which fit district instructional objec- 
tives, that is 

- developing or reviewing districts' scope and sequence 
of objectives 

- analyzing available tests against district's instruc- 
tional objectives by grade level, topic, etc. 

- selecting the best match of commercially available tests 

- developing CRT's 
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using student achievement test scores to target teachers for 
training, that is 

- analyzing student performance by teacher/by subject/ 
by grade level 

- increasing teachers* skills in student-deficient 
areas through staff development 

monitoring teaching by principal or by staff develop- 
ment specialist 

® developing remedial summer, pullout, or in-class programs for 
students 

- identifying student deficiencies 

- target teaching towards skills as required by test 

Other districts were more comprehensive in how they stated their 

purpose. They defined the purpose of their system as INSTRUCTIONAL 

MANAGEMENT. Their idea was to assert district responsibility for student 

learning by making learning the focus of all district operations. Some of 

the following activities to link testing and instruction were undertaken: 

^ developing a consensal district philosophy about educational 
goals, about quality educational environments, about methods 
of teaching 

communicating this philosophy to boards, advisory groups; 
informing and training administrators, teachers, in district 
philosophy and procedures 

^ establishing supervisory relationships between central office 
and schools and within schools to ensure compliance 

coordinating texts, tests, teaching at the school level 

^ coordinating curriculum, instruction, supervision, evaluation, 
and staff development at the central office level 

Extent of Coordination: Partial or Integrated 

Districts differed from one another in the complexity of their 

coordination arre;igements. 
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PARTIAL COORDINATION was characterized by some or all of the 



informal links among a few people. E.g., teachers received 
test scores with or without principal intervention; with or 
without central office intervention; and decided themselves 
what use to make of scores in relation to individual students, 
groups of students or the entire class. 

formal links among a few people, e.g., central office, 
principals, teachers, were assigned responsibilities in rela- 
tion to dissemination of, communication about, and action on 
test scores. 

On an ad hoc or routine basis, pairs or small clusters of cen- 
tral office staff met to discuss district -wide implications. 

On an ad hoc or routine basis, school principals met to 
discuss district -wide implications. 

On an ad hoc or routine basis, individual school staffs met to 
discuss school -wide implications. 



INTEGRATED COORDINATION was characterized by some or all of the 



Within central offices . Central office staff responsible for 
testing, evaluation, instruction, curriculum, supervision, 
staff development met regularly to analyze student achievement 
as assessed by test scores and other methods. They also 
reviewed district policy, planning, practice. 

Within schools . School staffs, advisory groups met regularly 
to review test scores and other evaluative data to develop 
school level approaches to problems or areas of concern. 

Between central offices and schools : 

- links between central offices, schools and classrooms in 
regard to Instruction were formal with clearly defined job 
responsibilities and accountability. 

- many ad hoc and permanent meetings -- principal's councils, 
advisory groups, task forces, met to consider instructional 
Improvement 

- colloquial atmosphere and informal relationships leads to 
exchange of Information 



f ol 1 owl ng : 



following: 
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Locus of Decision-making; District^ School > Classroom 

The foregoing discussion of purpose and coordination alluded to 
several different levels at which the basic decisions were made about 
translating the analyses of test scores into changes in instructional 
activities. Our sample districts arranged their systems so that critical 
decisions were made either at the central office level, at the school level 
or the classroom level. 

CENTRAL OFFICE decision-making was characterized by one or more of the 
following : 

district-level construction of grade-level objectives by 
subject areas 

district-level construction of criterion-referenced tests 

^ district-wide selection and use of a norm-referenced test 

district-wide evaluation of instruction through classroom 
visits, surveys, etc., of instructional efforts 

• district-wide formulation and conduct of staff development 
programs 

^ district-wide selection of texts to match tests 

SCHOOL LEVEL decision-making was characterized by one or more of the 
following: 

school -level planning teams, teacher/parent/community 

^ school -level receipt of information about student outcomes 
from either teacher-created tests, teacher-option CRT's, 
district-wide mandated CRT's; district-wide norm-referenced 
tests 

® school-level conduct of instructional evaluation via super- 
vision, teacher self-reports, district evaluations, outside 
evaluations 

school-level planning for school -year activities 
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^ school -level allocation of services, both personal and 
financial, to support local plans 

® school-level decisions about texts 

CLASSROOM LEVEL decision-making was characterized by one or more of 
the following: 

® individual teacher receipt of information about student 
learning, e.g., norm-referenced test scores, CRT's, teacher 
observations, text tests, student assignments, etc. 

® Individual teacher decisions about grouping, remediation, 
enrichment, alternative instruction, etc. 

• Individual teacher participation in professional development 

activities 

CONCLUSIONS 

As we have noted previously, our research is limited to six "heroic" 
districts that are trying to link testing, evaluation and instruction. 
These districts were not selected randomly. Thus the reader will appre- 
ciate the cautionary nature of the conclusions that follow. These conclu- 
sions are offered in the spirit of trying to anticipate concerns or ques- 
tions of evaluators or school district decision makers who might be consi- - 
dering developing or further implementating some effort to link district 
testing, evaluation and instruction. We will briefly discuss five conclu- 
sions that we hope will be helpful. 

Conclusion 1 . Perhaps the most important conclusion is that some dis- 
tricts have demonstrated that building such a t/e/1 system can be done and 
it can have a beneficial effect on the district. The districts differed 
with regard to how far thesy had progressed and how satisfied they were; 
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some had made considerable progress resulting in a variety of benefits, 
e.g., increased pupil performance, staff feelings of efficacy, greater 
sense of organizational purpose and direction. Because we have not con- 
ducted longitudinal studies we can reach no conclusions about the perma- 
nency of these t/e/i systems; however, the systems in operation for the 
longest time had developed a sizable "critical mass" of staff who were 
trained and experienced in using the program. Presumably they can resist 
any immediate attempts to substantially reduce or prematurely change the 
t/e/i system. 

Conclusion 2. There is no one single method or design . The district 
systems differed considerably in such factors as: the role of inservice; 
types of testing; evaluation designs; delivery systems; centralization of 
decision making. The reasons for these differences are not entirely clear 
but they include such factors as: community and district traditions; ex- 
ternally mandated practices; university-influenced planning; and evaluator 
and administrator personal or professional biases. In the most successful 
districts, these elements had been fitted together into a logical system. 

Conclusion 3 . The sequence for developing and implementing a t/e/1 
system varied from district to district . That is, the process these dis- 
tricts developed for building such a system did not conform to standard 
linear planning models. The sequence usually involved an "idea champion" 
at some point in the district's history who saw that certain activities 
were in place or emerging. He or she then began shaping the evolutionary 
wave that was building. Often the "idea champion" selected or focused on 
some procedure or technique, e.g., inservice training, CRT program, and 
built the t/e/i linking system around that activity. 
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Eventually, others joined the effort and a system of coordination and 
shared decision making developed. Indeed, even in the most experienced 
districts the systems are continually evolving as districts are buffeted by 
exogenous shocks in the form of budget cuts, population shifts, desegrega- 
tion efforts. Given the presently turbulent nature of public schools, it 
seems likely that this evolutionary process will continue for some time. 
The whole process takes time. The modal time span from initiation to full 
operation of the t/e/i linking system in our districts was approximately 
eight years. 

Conclusion 4 . Given the generic conditions of schools we discussed 
above, school district linking of testing and evaluat ion with instruction 
is not a naturally occurring phenomenon. All of these districts have had 
and continue to suffer setbacks in their efforts. Districts need constant 
and vigorous leadership. What Is more, in many districts some members of 
the critical mass leave the district for various reasons. This requires 
that they and their ideas be replenished in some way. 

Conclusion 5 . Linking tests and evaluation with instruction is first 
and foremost an analytical problem. There Is no single way, there is no 
universal schema. Successfully Implemented systems appeared to be the re- 
sult of capturing what was available and folding and bending it so that It 
fit into the local situation. This isn't to say that system development 
was totally a reactive process; Indeed, the district leadership actively 
had to adjust local practices and procedures to the ideas of such a sys- 
tem. All of this took thought and a realistic appraisal of local condi- 
tions, local traditions, local personnel, and local resources. But first 
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step had to be understanding and analysis. Such analytical processes were 
the necessary but not sufficient condition, for building and implementing 
an effective t/e/i linking system. 
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